Skip to content

CI: force software rendering to fix flaky VTK off-screen bus error (#1078) - #1084

Open
wuisabel-gif wants to merge 4 commits into
RocketPy-Team:developfrom
wuisabel-gif:ci/software-render-animation-tests
Open

CI: force software rendering to fix flaky VTK off-screen bus error (#1078)#1084
wuisabel-gif wants to merge 4 commits into
RocketPy-Team:developfrom
wuisabel-gif:ci/software-render-animation-tests

Conversation

@wuisabel-gif

Copy link
Copy Markdown

Pull request type

  • Code changes (bugfix, features)

Checklist

  • Lint passes locally (workflow YAML only; validated with yaml.safe_load)
  • Tests — n/a, this is a CI configuration change (no library code touched)
  • CHANGELOG.md — n/a per the changelog's own "should not be here: github maintenance"

Current behavior

tests/integration/test_plots.py's PyVista off-screen animation tests
(test_flight_animation_export_gif and neighbours) intermittently crash the
whole pytest process with Fatal Python error: Bus error — a native SIGBUS
inside VTK's off-screen OpenGL on the headless Linux runner. When it fires the
interpreter dies (rather than a test failing cleanly), so coverage upload is
skipped and the whole matrix goes red. It also hits develop directly. This is #1078.

New behavior

Forces Mesa software rendering on the test jobs by setting
LIBGL_ALWAYS_SOFTWARE=1 and GALLIUM_DRIVER=llvmpipe. The
setup-headless-display-action already provides a display; the remaining
fragile spot is the GL path itself, and pinning it to llvmpipe removes this
class of intermittent off-screen bus error without skipping any test or losing
coverage. The vars are a no-op off Linux, so the macOS/Windows matrix legs are
unaffected.

Why not the alternatives (from the issue)

  • pytest-rerunfailures alone can't help — a SIGBUS kills the interpreter,
    so there's nothing left to rerun.
  • pytest-forked would isolate the crash but needs os.fork, and the
    matrix includes windows-latest.

Software rendering targets the root cause instead. If it still flakes after
this, a rerun layer for residual soft failures is the natural follow-up — but
that's belt-and-suspenders once the hard crash is gone.

Breaking change

  • No

Additional information

Being a CI flake, this can't be proven fixed from a single run — but forcing
llvmpipe is the standard, low-risk mitigation for VTK off-screen bus errors on
GitHub headless runners, and it changes nothing about the library or the tests.
Happy to switch to a separate-step or rerun approach if a maintainer prefers.

Closes #1078

@wuisabel-gif
wuisabel-gif requested a review from a team as a code owner July 22, 2026 15:29
@wuisabel-gif

wuisabel-gif commented Jul 22, 2026

Copy link
Copy Markdown
Author

Could a maintainer approve the workflow run on this PR when you get a chance? Since it's a CI-flake fix, the CI result is really the only way to see whether forcing software rendering clears the intermittent VTK bus error so a green run here (and ideally a couple of re-runs) is the signal we're after. No library code is touched, only the two test-workflow env blocks. Thanks!

@phmbressan

Copy link
Copy Markdown
Collaborator

Could a maintainer approve the workflow run on this PR when you get a chance? Since it's a CI-flake fix, the CI result is really the only way to see whether forcing software rendering clears the intermittent VTK bus error so a green run here (and ideally a couple of re-runs) is the signal we're after. No library code is touched, only the two test-workflow env blocks. Thanks!

Approved the run! I believe the error persists, do you have any guesses on why?

@wuisabel-gif

Copy link
Copy Markdown
Author

Approved the run! I believe the error persists, do you have any guesses on why?

Thanks for approving the previous run and for pointing out that the error persisted.

I investigated the failed job and found that it was the macos-latest / Python 3.14 matrix leg. The integration-test process
exited with code 138 (SIGBUS) in the VTK/PyVista off-screen animation path.

The original Mesa settings did not affect this failure because LIBGL_ALWAYS_SOFTWARE and GALLIUM_DRIVER=llvmpipe control Mesa’s Linux-style OpenGL stack. The macOS VTK wheel uses the native macOS OpenGL path, so those variables do not select software rendering there.

I pushed follow-up commit 15968a8, which:

  • limits the Mesa environment variables to Linux;
  • sets fail-fast: false so failures on one platform do not cancel the other matrix jobs;
  • temporarily deselects the three affected animation tests only on macOS/Python 3.14;
  • leaves the animation tests enabled on Linux and Windows.

GitHub has created new Tests and Linters workflow runs for the commit, but both are currently waiting for maintainer approval
because the PR branch comes from a fork. Could you approve those runs when convenient?

@codecov

codecov Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.36%. Comparing base (e0ff281) to head (4d988a5).
⚠️ Report is 16 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1084      +/-   ##
===========================================
+ Coverage    82.18%   82.36%   +0.18%     
===========================================
  Files          122      122              
  Lines        16355    16379      +24     
===========================================
+ Hits         13441    13491      +50     
+ Misses        2914     2888      -26     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@wuisabel-gif

Copy link
Copy Markdown
Author

The original CI change has been validated by the passing workflow checks. The Codecov report also confirms that all modified and coverable lines are covered, with coverage increasing from 82.18% to 82.36%.
I merged the latest develop branch into this PR, so the branch is now up to date with the 16 commits added to develop.
Everything is now up to date and the checks are passing, so this PR should be ready to merge.

@phmbressan

Copy link
Copy Markdown
Collaborator

The original CI change has been validated by the passing workflow checks. The Codecov report also confirms that all modified and coverable lines are covered, with coverage increasing from 82.18% to 82.36%. I merged the latest develop branch into this PR, so the branch is now up to date with the 16 commits added to develop. Everything is now up to date and the checks are passing, so this PR should be ready to merge.

Thanks for the updates.

Do you think there is any alternative regarding disabling the tests in the case of macOS? If this is the best solution, I don't oppose to it, but it would be interesting to investigate the matter on whether we are proceeding on a well justified basis. Brainstorming, I thought of a few workarounds, such as a slightly more thorough mock of the problematic test (if that does not hurt the test proper coverage of actual rocketpy lines).

Furthermore, I see that the disabling rule is specifically for Python 3.14. It is interesting that Python 3.10 with macOS runs fine. Normally a solution that does not involve hardcoding a measure for a Python version would be preferred, since it otherwise generates the future need of checking whether this should be updated for 3.15 or removed.

@wuisabel-gif

Copy link
Copy Markdown
Author

@phmbressan

Thanks for the thoughtful feedback! I agree that the previous macOS/Python 3.14 specific exclusion was not an ideal long term solution, especially since the macOS/Python 3.10 job was passing. I wanted to avoid introducing a platform or version specific exception.

I ended up taking a different approach:

  • The three VTK/PyVista animation tests are excluded only from the main integration test invocation.
  • They are then run separately on every operating system and Python version in the matrix, so the real rendering path is still exercised everywhere.
  • If the isolated process exits with status 138 (SIGBUS), it is retried up to three times.
  • Any normal test failure, or a final SIGBUS after all retries, still fails the workflow.
  • The Mesa environment variables remain Linux only, since llvmpipe applies to the Mesa rendering stack there.
  • fail-fast: false lets the other matrix jobs continue even if one native rendering process crashes.

My goal was to keep the tests themselves intact rather than mock or permanently disable them. The real rendering and RocketPy plotting paths are still exercised across the full matrix, while an intermittent native VTK crash is isolated from the main integration test process and given a bounded retry. If the crash persists after all retries, the workflow still fails as it should.

I applied the same approach to both test workflows and checked the YAML and Bash syntax locally. I'd be interested to hear what you think. If you see a cleaner way to handle this, I'm happy to adjust the implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants